Device and method for the processing of image data

ABSTRACT

A source image (S 1 ) distorted by a camera lens can be transformed into a rectified target image (T), by means of a e tabular imaging specification. The above occurs during read-out from the image sensor and in real-time. Each source pixel in the source image is assigned none, one or several target pixels in the target image (T 1 ). A first controller (C 1 ) controls the image sensors (B 1 , B 2 ) accurately with tine and the image equalisation and image correlation. A second controller (C 2 ) controls the first controller (C 1 ) and works in a manner temporally decoupled from the above.

CLAIM FOR PRIORITY

[0001] This is a national stage application of International ApplicationNo. PCT/DE01/02015, which was published in the German language on Nov.28, 2002 and which was filed in the German language on May 25, 2001.

TECHNICAL FIELD OF THE INVENTION

[0002] The invention relates to an arrangement and method for processingimage data, particularly in imaging systems for vehicle occupantprotection systems.

BACKGROUND OF THE INVENTION

[0003] Microsoft Research Technical Report MSR-TR-98-71 “A Flexible NewTechnique for Camera Calibration” discloses a method for compensatingimage distortions whereby a mathematical rule is used to map a sourceimage recorded by a camera onto a target image. The computational rulecalculates the corrected target image from the source image loaded intomain memory.

[0004] In occupant protection systems, exacting requirements are placedon the speed of optical image recognition, as the position of a personon a vehicle seat must be rapidly established in the event of anaccident in order to deploy the restraint system accordingly. The imagesensors provided in the camera of an imaging system record images of animage area in close succession, the resulting image data of an imagehaving to be read out of the image sensor by a control unit before thenext image is recorded.

[0005] This requires a large amount of memory for storing the image dataof an image, precise timing for transmitting the image data to thecontrol unit and considerable computing power for further processing ofthe images.

SUMMARY OF THE INVENTION

[0006] An object of the invention is therefore to cost-effectivelyreduce the processing overhead for image data.

[0007] According to an aspect of the invention, a method forcompensating image distortions is provided which can be usedparticularly in imaging systems for occupant protection systems. Animage distorted by the optics of a camera system produces, in the camerasystem's image sensor, a source image which is distorted in differentways depending on the quality of the optics, the focal length of thecamera system and other optical parameters. The source image ispreferably broken down into individual source pixels each disposed at apredefined position in the source image and whose grayscale valuesrecorded by the image sensor are in each case stored under a predefinedsource pixel address in the image sensor.

[0008] The source image is preferably mapped into a target image via apredefined mapping rule, whereby a corrected target image is produced.The target image preferably includes target pixels whose grayscalevalues are stored in each case under a target pixel address in a targetmemory, a source pixel being mapped into no target pixel or into one ormore target pixels, the grayscale value of the source pixel addressbeing transferred to the target pixel address.

[0009] The mapping rule for correcting a source image to produce atarget image is preferably stored in tabular form in a rectificationtable in a memory of a first control unit. The first control unit alsotakes over the complex and time-precise control of the image sensor(s),thereby advantageously enabling the mapping rule to be quicklyprocessed. In addition, it is unnecessary to buffer the source image,thereby saving considerable memory space.

[0010] This advantageously reduces the required memory space andsimultaneously enables correction of the source image to be performedwithout delay, which is particularly necessary for occupant protectionsystems.

[0011] Mapping of the source image into the target image according tothe specified mapping rule produces a target image having fewer pixelsthan the source image. There are therefore a number of source pixelswhich are not mapped into the target pixel. In addition, the imagesensor generally captures more information than is actually required.This redundant information is filtered out by the mapping rule.Filtering and data reduction are therefore advantageously performed.Only the target image generated by the mapping rule is stored in thefirst control unit, which means that memory space is in turn saved inthe first control unit.

[0012] Two image sensors are preferably connected to the first controlunit. The corrected target images are preferably correlated row-wisewith one another in the first control unit to produce a range imagecontaining not only grayscale value information but also rangeinformation of the relevant image points of the camera. From the imagedata of the two target images, only part of the range image ispreferably formed, buffered, and fed out cyclically or when requested bya second control unit, thereby advantageously saving memory space.

[0013] The first control unit controls the image sensor(s), providestiming matched to the image sensor and basically performs all thetime-critical and compute-intensive image processing operations. Theresulting image data is fed out to a second control unit via a specifiedinterface, preferably a standard interface such as PCI, local bus etc.The second control unit takes over the computation results of the firstcontrol unit, e.g. the range image or parts of the range image, andcontrols the first control unit. In the second control unit, thereceived image data is analyzed using seat occupancy classificationalgorithms. It is possible to detect, for example, the position of anoccupant on a vehicle seat, the position of the occupant's head, theposition of a child seat or an unoccupied vehicle seat. The resultingdata is forwarded to an airbag control unit (ACU).

[0014] The camera optics of an image sensor are subject to manufacturingtolerances. To compensate for the manufacturing tolerances, therectification table associated with the given camera optics ispreferably determined at the end of the production line by buffering theimage data of a reference image acquired by one of the image sensors ina first memory. This first memory can be in the first or the secondcontrol unit. Using an initialization routine, the appropriaterectification table is created and stored in this first memory so thatthe storage space of the first memory is advantageously used twice. Thisinitialization routine is preferably executed in the second controlunit. The rectification table is stored in a read-only memory of thefirst control unit. Alternatively, the rectification table is stored inthe read-only memory of the second control unit, e.g. a flash memory,and transferred to the first control unit at startup.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 shows the interior of a vehicle with an optical imagingsystem;

[0016]FIG. 2 is a block diagram of an arrangement for image processing;

[0017]FIG. 3 is a flowchart of an initialization routine forcompensating for optical system tolerances; and

[0018]FIG. 4 is a flowchart of an image processing routine.

DETAILED DESCRIPTION OF THE INVENTION

[0019]FIG. 1 schematically illustrates a vehicle 1 in which there ispreferably located a vehicle seat 2 having a seat pad 23, a backrest 21and a head restraint 22 mounted thereon. In the lining of the vehicleroof 3 there is disposed, preferably between the two front seats, anoptical camera system 7, 71, B1, B2, C1,C2 with which a predefined imagearea Bi of the vehicle interior can be captured. Preferably two imagesensors B1, B2 cover the image area Bi comprising the vehicle seat 2with any subject 9 located thereon via a camera optical system. In FIG.1, the subject 9 is schematically illustrated as a vehicle occupant.

[0020] In further embodiments the subject 9 can be a child seat, objectsor similar, or the vehicle seat 2 can be unoccupied.

[0021] In the front part of the vehicle 1, under the windshield 4, thereis disposed a dashboard 5 below which there is a footwell 8 for the feetand legs of the occupant 9 and in which an airbag 26 is located. Thelower extremity of the footwell 8 is delimited by the vehicle floor 6 onwhich seat rails 24 are disposed. In the area of the lower part of theseat pad 23, the vehicle seat 2 is connected to the seat rail 24 viasupports. The vehicle seat 2 is therefore displaceably disposed in theX-direction, i.e. the vehicle direction.

[0022] The camera system 7 preferably comprises two image sensors B1,B2, a light source 71 preferably equipped with a plurality oflight-emitting diodes or at least one laser diode, and an analysis unitC1,C2. The image area Bi is illuminated both by the light source 71 andby any available ambient light. The optical axes of the two imagesensors B1, B2 have a predefined spacing L. This enables rangeinformation of the subjects in the predefined image area Bi to thecamera system 7 to be acquired from the images captured by the two imagesensors B1, B2 using stereo image processing methods. The camera 7preferably incorporates the two image sensors Bi, B2 and the lightsource 71 in a compact housing. The analysis unit C1, C2 is likewisepreferably disposed in the same compact housing, as the volume of datatransmitted by the image sensors B1,B2 to the analysis unit C1,C2 ishigh. The exemplary image sensor B1 preferably has a matrix-shaped pixelarrangement with a resolution of 320×288 pixels and a grayscale depth orgrayscale resolution of 8 bits=256 grayscale values per pixel. Using twoimage sensors B1 and B2 and a minimum sampling rate of 50 images persecond per image sensor results in an overall data transmission ratebetween the image sensors B1,B2 and the analysis unit C1,C2 of

320×288×8×2×50=73.728 Mbit/s.

[0023] In another embodiment, only one image sensor B1 or B2 isprovided, thereby reducing the costs. Here, the required rangeinformation is preferably obtained from optical delay measurements orother image processing methods.

[0024]FIG. 2 shows the block diagram of an image processing arrangement.Two image sensors B1 (left) and B2 (right) each capture an image area Bivia an optical system OPT 1, OPT 2. As essentially identical processesoccur in the two image sensors B1, B2, the image processing operationwill now be described using the example of the left image sensor B1.

[0025] The image to be captured of the image area Bi is distorted by theoptical system OPT 1 with the result that a distorted source image S1 isproduced in the image sensor B1. The image sensor B1 is preferablycontrolled by a first control unit C1. A sensor timing unit T1 in thefirst control unit C1 supplies the necessary control signals preciselytimed for the image sensor B1. The source image S1 captured by the imagesensor B1 must be read out within a short time, e.g. at a sampling rateof 50 images per second in a few milliseconds. In addition, because ofthe analog design of the image sensor B1, the storage time of a sourceimage S1 in the image sensor B1 is short.

[0026] The image data present in the image sensor B1 is transmittedpixel by pixel to the first control unit C1, a pixel at a predefinedpixel address containing a grayscale value. The image data supplied bythe image sensor B1 is processed by a rectification controller C13 inthe first control unit C1. The rectification controller C13 controls thecorrection of the source image S1 to produce a target image T1. Thesource image S1 is essentially mapped into a target image T1 pixel bypixel using a rectification table TA stored in a memory M10. Thecorrected (rectified) left target image T1 and the corresponding righttarget image T2 are stored in a buffer (target memory) M11, M12 in thefirst control unit C1. A census transformer C11 reads out at least partsof the two target images T1, T2, processes them and correlates the partsof the left and the right target image T1, T2 with one another to obtainrange information of the captured image.

[0027] The correlation is preferably performed in a correlator C12 towhich 6 preprocessed rows of the left target image T1 and 6 preprocessedrows of the right target image T2 are fed. The range image AB which hasbeen correlated and provided with range information is stored in amemory M0. Preferably only a few rows of the correlated image are storedor transformed. A central control unit C10 located in the first controlunit C1 controls all the functional blocks T1, C13, MUX1, MUX2, C11, C12contained in the first control unit C1, and the memories M10, M11, M12,M0. Upstream of the target memories M11, M12 and the memory M10 thereare provided multiplexers MUX2, MUX1 with which the central control unitC10 controls the memory accesses to the individual memory areas.

[0028] The central control unit C10 is preferably controlled by a secondcontrol unit C2. The second control unit C2 is largely exempt from thetime-critical requirements for reading out the image sensors B1, B2 andsubsequent rectification and correlation of the image data and istherefore time-decoupled. Consequently, the control unit C2 can reactflexibly to external events initiated e.g. by an airbag control unit C3connected via an interface. The second control unit C2 is equipped witha main memory M2 and a nonvolatile memory M3. At the request of thesecond control unit C2, the corrected and correlated image data storedin memory M0 of the first control unit C1 is preferably transferred tosaid second control unit. In addition, the second control unit C2supplies the system clock and transmits commands (Execute) to thecentral control unit C10 of the first control unit C1. The image datatransferred by the memory M0 is further processed in the second controlunit C2. In the second control unit C2, a pattern recognition algorithmis executed by which the occupancy state of a vehicle seat is classifiedfrom the image data.

[0029] Advantageously, because of the memory M10, M11, M12 present inthe first control unit C1, no external memory with a correspondingnumber of required lines is necessary.

[0030]FIG. 3 shows the flowchart for initializing an image processingarrangement. The optical systems OPT1 and OPT2 are to be manufactured asinexpensively as possible, resulting in high manufacturing tolerances.As a result, each optical system OPT1, OPT2 is subject to differentdistortions. Using the initialization routine described below, arectification table TA pertaining to the relevant optical system iscreated for each optical system at the end of the production line. As aresult it is advantageously possible to compensate for even highmanufacturing tolerances of an optical system type series.

[0031] At the start of the initialization routine, a reference image RBis held in a predefined position in front of the optical system OPT1 ofthe image sensor B1. The reference image RB exhibits a predefinedpattern, e.g. vertical and horizontal lines L2,L1 and/or dots P eachoccupying a predefined position. The image sensor B1 now captures thereference image RB, thereby producing a distorted reference image, e.g.the source image S1 in the image sensor B1. The image data assigned tothe source image S1 is read out by the first control unit C1 and storedin the memory M10. Using a predefined computational rule, the firstcontrol unit C1 determines the rectification table TA from the imagedata and stores it in the memory M10 or in the read-only memory M13 ofthe second control unit C2. The tabular data of the rectification tableTA is subsequently copied to the memory M10 at initialization, e.g. whenthe occupant protection system is activated.

[0032] In a further embodiment, the computational rule to determine therectification table TA is executed in the second control unit C2. Thisis possible, as the creation of the rectification table TA takes placeat the end of the production line and is therefore not time-critical.

[0033] The rectification table TA is now available in a read-only memoryM3. Initialization is therefore complete.

[0034]FIG. 4 shows the flowchart of an image processing routine. At thestart of the routine the rectification table TA is loaded from theread-only memory M3 of the second control unit C2 into the memory M10 ofthe first control unit C1, the rectification table TA being exemplaryfor the processing of a source image S1 of the image sensor B1. Arectification table is preferably provided for each image sensor.

[0035] The first control unit C1 reads the image data of the distortedsource image S1 out of the left image sensor B1 pixel by pixel. Usingthe mapping rule stored in the rectification table TA, the data ismapped pixel by pixel into a corrected target image T1 in therectification controller C13 of the first control unit C1. The correctedtarget image T1 is stored in the memory M1. The image data of thedistorted source image S2 is processed correspondingly. The resultingtarget image T2 is stored in the target memory M12.

[0036] The image data of the target images T1, T2 are preferably readout row-wise from the memories M11, M12 and processed using a predefinedcensus transform to produce left and right census rows, preferably sixfor each left and right image, which are buffered and correlatedrow-wise with one another. The image data of the pixels of thecorrelated rows additionally contains range information and is stored inthe memory M0. On request, this image data is transferred to the secondcontrol unit C2 which now classifies the transferred image data usingpattern recognition algorithms. The classification result is transmittedto the airbag control unit C3 (ACU).

[0037] The first control unit C1 is preferably implemented as an ASIC(Application Specific Integrated Circuit) or FPGA (Field ProgrammableGate Array). The second control unit C2 is preferably implemented as amicrocontroller or microprocessor. The first and the second control unitC1, C2 can be incorporated in one housing and interconnected viaconductive tracks. In a further embodiment, the first and the secondcontrol unit C1, C2 can be integrated in a package or even on a chip. Inthe second control unit C2, triggering decisions for occupant protectionsystems can be additionally implemented.

[0038] In the first control unit C1, a large number of operations areperformed in parallel, whereas in the second control unit C2 only asmall number of operations or a single operation are processed inparallel.

1. An arrangement for processing image data in imaging systems foroccupant protection systems, comprising: a first control unit which, inreal time, reads out from at least one image sensor, grayscale values ofthe image data as source image pixels; an optical system which distortsthe image data, where the first control unit corrects the image datadistorted by the optical system using a tabular mapping rule so that atarget image is produced; and a second control unit for controlling thefirst control unit and taking over the target image data processed bythe first control unit, the second control unit providing the firstcontrol unit with range information and forming seat occupancyclassification, wherein by mapping the source image into the targetimage according to the mapping rule, a target image having fewer pixelsthan the source image can be obtained.
 2. The arrangement according toclaim 1, wherein the first control unit contains a rectification tablestored in a first memory and which can be used to execute the mappingrule for correcting the image supplied by the image sensor.
 3. Thearrangement according to claim 2, wherein to initialize therectification table, image data of a reference image captured by one ormore image sensors is stored in the first memory.
 4. The arrangementaccording to claim 1, wherein the first control unit is implemented asan ASIC or FPGA, the second control unit is implemented as amicrocontroller or microprocessor, and the first and the second controlunits are disposed separately or one of in an IC package and on a chip.5. The arrangement according to claim 1, wherein two image sensors arepresent, the first control unit correlating the corrected image data ineach case to produce a range image.
 6. The arrangement according toclaim 1, wherein at least one image sensor is linked via the firstcontrol unit to the second control unit.
 7. A method for processingimage data is in imaging systems for occupant protection systems,comprising: reading the image data available in an image sensor of asource image distorted by an optical system from the image sensor by afirst control unit in which the image data is corrected in real timeusing a tabular mapping rule so that a target image is produced, wherebyin a second control unit which controls the first control unit, thecorrected target image data is provided with range information andclassified, wherein by mapping the source image into the target imageaccording to the mapping rule, a target image having fewer pixels thanthe source image can be obtained.
 8. The method according to claim 7,wherein a pixel under a predefined pixel address contains a grayscalevalue.
 9. The method according to claim 7, wherein the target image orthe image data of the target image is buffered in a memory of the firstcontrol unit.
 10. The method according to claim 7, wherein if a firstand a second image sensor are present, the two corrected target imagesproduced are correlated row-wise with one another to determine rangeinformation, and the resulting image pixels or image data of the imagepixels provided with range information disposed in at least one imagerow, are buffered in a memory of the first control unit, and at therequest of a second control unit, the buffered image pixels or imagedata of the image pixels are transferred to the second control unit forfurther processing, vehicle seat occupancy classification beingperformed in the second control unit.
 11. The method according to claim10, wherein the result of the vehicle seat occupancy classification isforwarded to a central occupant protection control unit.
 12. The methodaccording to claim 10, wherein a predefined reference image is capturedby at least one image sensor to initialize or create a rectificationtable, the captured reference source image is read out of the imagesensor and stored in a first memory, the rectification table is createdfrom the stored reference source image, and the created rectificationtable is stored in the first memory, whereby the reference source imagestored in the first memory is at least partially overwritten by therectification table.
 13. The method according to claim 12, wherein thereference source image has a predefined pattern.
 14. The methodaccording to claim 12, wherein the reference source image is captured bythe two image sensors and used as reference for the correlation.
 15. Themethod according to claim 12, wherein a rectification table is providedfor each image sensor or one rectification table is available for allthe image sensors.
 16. The method according to claim 7, wherein a systemclock of the second control unit is essentially independent of a systemclock of the first control unit.
 17. The method according to claim 7,wherein image processing in the first control unit is executed inparallel with classification in the second control unit.
 18. The methodaccording to claim 7, wherein the first and the second control unitoperate largely independently of one another.